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Abstract 


Quality and accountability mandates require institutions to monitor online instruction in 
a uniform and complete manner. In many institutions, instructor training is sparse and 
faculty evaluation occurs only through end-of-course student evaluations that may or may 
not yield adequate information on how the instructor performs online. Consequently, the 
online instructor evaluation system (OIES) was developed to ensure the finest quality edu¬ 
cational experience for online students via a systematic approach to faculty training, men¬ 
toring, and evaluation. Research has shown that combining mentoring and evaluation is 
not feasible, and therefore another approach is warranted. 
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Introduction 


Over the past decade, most colleges and universities in the United States have experienced 
a dramatic increase in the growth and popularity of online degree programs. According to 
research conducted by the Sloan Consortium, distance learning is growing rapidly, with 
83% of higher education institutions offering some form of distance learning (Allen & Sea¬ 
man, 2008). Similarly, community colleges report an 11.3% increase in distance education 
enrollments, a figure substantially ahead of overall national campus enrollments, which aver¬ 
aged less than 2% (Lokken, 2009). Further, in 2008 an overall 12.9% growth in online learn¬ 
ing in higher education occurred, exceeding a 1.6% growth in traditional classes during the 
same period (p. 5). Online courses clearly are entrenched in modern higher education when 
measured by the volume of courses, number of faculty and students involved, infrastruc¬ 
ture investment dollars, or other parameters. 
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Students cite convenience and flexibility as dominant reasons to enroll in online courses 
(Northrup, 2009). Faculty also benefit from online course flexibility and convenience as it 
provides more professional options such as teaching part-time at one or more institutions, 
an option to supplement academic retirement, and professional development opportuni¬ 
ties. Institutions benefit from enhanced access and revenue from students who reside a 
great distance from the institution’s geographic location. Employers appreciate the avail¬ 
ability of workers with additional qualifications, attained with less absenteeism or career 
interruption. 

As institutions of higher learning strengthen their infrastructures to accommodate the de¬ 
mand for online courses and programs, urgent needs for trained and properly motivated 
faculty emerge. Through proactive measures to train, mentor, evaluate, and remediate on¬ 
line faculty, colleges and universities can limit potential student problems and complaints. 
It is not sufficient merely to train instructors without conducting follow-up administrative 
or peer scrutiny of their performance. Quality and accountability mandates obligate in¬ 
stitutions to monitor online instruction in a uniform and complete manner. Institutional 
circumstances require that such monitoring be conducted efficiently. 

Park University, with a historic campus located in Parkville, Missouri (near Kansas City) 
and 40 campus centers in 21 states, has developed and implemented a quality management 
system to ensure that students are taught by trained, mentored, and evaluated faculty. Park 
has experienced a surge in distance education enrollments, with a student enrollment full¬ 
time equivalent (FTE) of 20,000 and over 50,000 total enrollments annually. The univer¬ 
sity employs approximately 350 online instructors to teach more than 450 course sections 
each eight-week term in order to accommodate student demand. Because Park is commit¬ 
ted to high standards, academic integrity, course content consistency, and effective mea¬ 
sures of learning outcomes, the transition to the online course delivery mode has necessi¬ 
tated new approaches to monitoring and evaluating academic quality. 

In many institutions, instructor training is sparse and faculty evaluation occurs only 
through end-of-course student evaluations that may or may not yield adequate information 
on how the instructor performs online. However, Park promotes academic rigor by tun¬ 
neling substantial resources into faculty training and evaluation. Consequently, the online 
instructor evaluation system (OIES) was developed to ensure the finest quality educational 
experience for online students via a systematic approach to faculty training, mentoring, and 
evaluation (Mandernach, Donnelli, Dailey, & Schulte, 2005). Using a case study approach, 
the OIES was created based upon institutional need, existing research on online learning, 
and resources available for instructor mentoring and evaluation. 
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Literature Review 


Background 

Institutions with online learning courses and programs are understandably interested in 
best practices and empirical information that can strengthen their distance learning opera¬ 
tions. Consequently, evaluation of online courses is a popular topic in the research litera¬ 
ture (Dykman & David, 2008; Lord, 2009; Mandernach et.al., 2005; Weschke & Canipe, 
2010; Villar Angulo & Alegre de la Rosa, 2007; Avery, Bryant, Mathios, Kang, & Bell, 2006). 
Distance learning practitioners have struggled to create effective models for designing, 
assessing, and evaluating online courses. Observation of the Web sites of online and/or 
higher education associations reveals various references to guidelines and best practices 
that encourage excellence in online learning. Individual authors also add to the depth and 
breadth of online learning interest. The oft-cited seven principles (Chickering & Ehrmann, 
1996) and subsequent application of the principles to online course evaluation (Graham, 
Cagiltay, Lim, Craner, & Duffy, 2001) are but two examples. Such seminal works, with their 
learning-focused criteria, provide the theoretical background of adult learning theory that 
guides this research. Additionally, these adult learning principles have guided the develop¬ 
ment of many online programs. 

The system described herein incorporates an array of best practices for teaching online, 
notably the seven principles (Chickering & Ehrmann, 1996; Graham et.al., 2001). Through 
our initial review of the literature, coupled with an analysis of key components of the Park 
course platform and course layout, an evaluation process was established using a case study 
approach. As the OIES is a dynamic system, this section presents some of the research that 
influenced this process. By implementing various versions of the document over a period of 
time and analyzing the outcomes, improvements were made to the evaluation tool and the 
overall process. While thematically the coverage areas remain the same, the evaluation tool 
process has been refined to clarify what the institution expects from instructors. The fol¬ 
lowing sections provide an overview of focal areas that emerged from the OIES and which 
contributed to the present streamlined model used at the university. 

Community in the Classroom 

As noted above, the seven principles (Chickering & Ehrmann,1996; Graham et al., 2001) 
provided the basis for the criteria established for the review. The importance of establish¬ 
ing community in the classroom has been confirmed by a number of writings. Dykman 
and Davis (2008) affirm the use of a personal profile as the first real opportunity to set the 
tone for the course with students. Moreover, it is an important early opportunity to con¬ 
nect with the students and therefore warrants careful consideration and preparation. The 
authors conclude that “Consistent interaction, steady participation, and timely reinforce¬ 
ment are the keys to keeping the students in an online course involved and active” (p. 287). 
Malbrito (2001) aligns with the preceding authors in recognizing the importance of student 
introductions in the course. Inducing students to post meaningful profiles is well worth the 
effort for the overall success of the virtual classroom experience. The OIES emphasizes the 
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importance of encouraging all students to post an introduction and suggests that instruc¬ 
tors should reach out to students who may be late in posting their introductions. The OIES 
also reminds instructors to acknowledge the presence of students in the online classroom 
and to comment in an authentic way on the student’s sharing of personal information. 

Discussion Facilitation in Online Instruction 

Another area of the OIES that emerged from the literature is effective discussion facilita¬ 
tion in online instruction. The OIES examines both policy compliance and level of engage¬ 
ment by the online instructor. Instructors are required to participate a specific number of 
days each week and to engage in critical thinking dialogue with students to promote quality 
postings. In support of this emphasis area, Dietz-Uhler, Fisher, and Han (2007) reviewed a 
study of retention rates in online education. Some students reported that their reasons for 
dropping out of a course included a lack of instructor participation, feedback, and replies to 
student e-mails. Barnard, Paton, and Lan (2008) suggest that instructors of online courses 
should be especially concerned with creating learning environments where positive percep¬ 
tions toward online course communication and collaboration can be informed and fostered. 
This latter qualitative study revealed that while positive instructor feedback and interest in 
their work elicited pleasure and pride in students, failure by the instructor to acknowledge 
their potential produced shame. These findings bear clear andragogic implications for on¬ 
line teaching. Ultimately the instructor must be willing to re-evaluate and to revise teacher- 
learner roles and relationships. Thus, through analyzing the relevant literature the evalu¬ 
ation team realized that discussion facilitation distinctly differentiates the online course 
from an independent study. Therefore, acknowledgement by instructor and peers is central 
to the learning success of each student and poses a unique opportunity for instructors. 

Mann (2005) describes discussion as an essential dimension of the online course. The dis¬ 
cussion feature should resemble a conversation that allows each participant a voice in the 
learning group and its workings. Responsibility is reciprocal. Mann suggests that instruc¬ 
tor withdrawal from the discussion will lead to stagnation. To avoid this, instructors must 
be fully cognizant of their privilege and power and use them judiciously to engage learners 
with their classmates and the content. This vital classroom communication dynamic is re¬ 
flected in the tone, engagement, and guidance provided in the online instructor observa¬ 
tion. It seeks to eliminate stagnation in favor of ongoing instructor presence and to prompt 
feedback to students throughout the course. These critical points are precisely captured in 
the OIES. 

Assessment, Grading, and Feedback 

Other areas of emphasis in the OIES include a focus on the instructor’s approach to course 
assessments, grading, and feedback. Dykman and Davis (2008) address this issue in their 
dialogue on efficacy in the online classroom. These authors confirm that students are al¬ 
ways very concerned about grades and that too much ambiguity about grading in an on¬ 
line course can destroy the instructor’s credibility with the students. The authors posit that 
without the normal contact found in a conventional classroom, students seldom know what 
to expect from an instructor. Ambiguity or inconsistency in grading quickly destroys trust. 


Vol 13 | No 3 


Research Articles 


June 2012 


86 






Creating a Sustainable Online Instructor Evaluation System: A Case Study Highlighting Flaws when Blending Mentoring and Evaluation 


Schulte, Dennis, Eskey, Taylor, and Zeng 


The OIES provides a rigorous analysis of the instructor’s progress in grading, use of rubrics, 
and providing individualized feedback to students on their performance. 

Some key findings that relate to the development of the OIES involve a 2010 survey of more 
than 550 higher education faculty and students in the United States and Canada. More 
than a third of the faculty in the sample said they were not proficient with their institution’s 
course management system. Students reported being either “pretty much lost” or that they 
“know a little, a few basics” (Primary Research Group, 2010). These findings support claims 
that it is essential for instructors to be supported in using the tools in order to provide feed¬ 
back and grading and also that students must be provided guidance that help them learn 
and to assimilate into the online culture. Through the self-assessment feature of the OIES, 
reflection and metacognition enable our instructors to consider how effective they are at 
providing feedback and assigning grades. Through this process instructors build their ca¬ 
pacity for integrating best practices in their work. 

Course Climate and Learning Environment 

A final best practice to be discussed here is one of the most critical focal areas of the OIES, 
course climate and learning environment. Over the years of implementing the OIES, our 
team has strived to assure that the learning climate in our online courses is conducive to 
the academic success of a diverse group of adult learners. Our experience with observing 
courses over the past several years affirms the research of Gilmore and Warren (2007). 
The findings of their qualitative study of online seminars confirm that when an instructor 
is absent or provides limited interaction in the online classroom, students feel isolated in 
their learning. Students are then forced to navigate the curriculum alone or to bond with 
classmates, who are not content experts. The OIES evaluators have recognized that ample 
clarification, addressing students by name, and timely follow-up to questions and concerns 
are beneficial in establishing student/instructor respect and trust in the classroom. Dyk- 
man and Davis (2008) attest similarly that consistent interaction, steady participation, and 
timely reinforcement are the keys to keeping online students involved and active. The OIES 
process affirms that quality online teaching requires extensive interaction and a substantial 
commitment of the instructor’s time and effort. 

This section highlighted some of the supportive literature and lessons learned as related to 
the OIES development. It should be emphasized that this OIES process was integrated in 
tandem with an academic institutional examination to assure a seamless investment in in¬ 
structor efficacy and engagement in the online classroom. Aspects of traditional and online 
evaluation processes in the field of faculty development were used to provide additional 
insights into the model, strategy, and implementation that would work well at the univer¬ 
sity and at the same time also provide a unique learning context. The OIES offers a rigorous 
evaluation system for instructors who may require extensive support in transitioning to the 
online instructional environment. 
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Effective Methods of Performance Management (Faculty Devel¬ 
opment) 

Lord (2009) suggests that professors should develop delicate ways to identify weaknesses 
and praise strengths. Weschke and Canipe (2010) describe an evaluation model similar to 
the OIES which is conducted peer-to-peer and is not derived from administrative leader¬ 
ship. This similar model identifies collaborations among instructors that lead to construc¬ 
tive faculty development. After several years of implementation, the results of many obser¬ 
vations suggest that our emphasis on carefully worded, concrete checklists has yielded less 
ambiguity and confusion about the performance level expected of online instructors. 

The strength of the OIES lies in its futuristic element of embedding opportunities for self- 
guided reflection and learning by the instructor being evaluated. According to Ciezki and 
Khare (2010), these types of self-directed reflections and participative reviews benefit both 
employers and employees as they build a sense of ownership and motivation through com¬ 
munication and negotiation. In many ways, self-directed learning allows employees to as¬ 
sume responsibility and a certain level of personal investment. This unique combination of 
reflection, dialogue, review of professional development topics, and final summary in the 
OIES provides a robust quality assurance process. It also provides a tier of support in work¬ 
ing with a faculty evaluator, thus humanizing the evaluation process and helping to align 
the instructor’s self-assessment with the actual criteria established on the OIES. Further¬ 
more, it provides an objective observation with rationale for best practices. 

A range of sensitive topics are addressed in the faculty evaluation. The OIES model bal¬ 
ances objectivity with a tightly written checklist and rubric that eliminate subjectivity. It 
has also fostered a broad-brush professional development approach for acquiring skills at 
teaching online. 


Case Study: An Overview of the OIES Method 


First implemented in 2004, the online instructor evaluation system aided the Park Uni¬ 
versity Distance Learning division in the areas of online instructor mentoring, evaluation, 
course scheduling priority, and professional development. A total of 437 separate OIES 
evaluations have been conducted on online instructors up to the present day. The number 
of evaluators has fluctuated from term to term depending upon evaluator availability and 
course numbers needed. For example, one evaluator might have conducted evaluations in 
one eight-week accelerated term, followed by a term in which five or six evaluators per¬ 
formed OIES evaluations. Over the years of use, 12 OIES evaluators have been trained and 
utilized. The current cadre of six evaluators includes members who have followed the pro¬ 
cess from its inception to the present. The materials that follow in this case study are based 
on the cumulative experiences of this author and evaluator workgroup. 

The robust and comprehensive nature of the OIES is one of its strengths. The OISE includes 
both formative and summative evaluation components. The formative reviews, a total of 
five, are completed by the instructor evaluators at the beginning of the term and continue 
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every two weeks during the eight-week term. Each review focuses on specific online best 
practices and/or Park University online learning policy. The reviews provide boxes for each 
criterion, the evaluator rating of the level to which the instructor has met the criterion, and 
comments. The comment box provides the most useful formative/mentoring feedback for 
online instructors. Evaluators have compiled banks of commonly used comments to facili¬ 
tate the completion of each review, but often it is necessary to customize the comments for 
the needs and idiosyncrasies of each instructor. 

The intended result of each formative review is to spark a mentoring dialogue between 
instructor and evaluator. Each formative review is posted to a secure online portal. An au¬ 
tomated email notifies the instructor that the review is available. Frequently, questions, 
suggestions, and guidance on best online teaching practices dominate the subsequent men¬ 
toring discussions. The OIES team has found that reactions to the formative reviews are as 
unique as the instructors. Some instructors become very involved in the “back and forth” 
discussions (via phone, email, or both) of the reviews and possible modifications of their 
teaching practices. Other instructors mistakenly view the formative reviews as “judgments” 
on their teaching. Dispelling these misperceptions quickly and efficiently is important so 
that mentoring discussions can prevail. 

In an effort to maximize the benefits of mentoring, the OIES includes an instructor self¬ 
review component during the formative stages. Self-reviews, which are completed every 
two weeks, mirror the corresponding weeks’ formative reviews. Criteria for the self-reviews 
and formative reviews are complementary so that both instructor and evaluator focus on 
the same items at once. While the self-reviews are not mandatory, they are strongly encour¬ 
aged for reasons of professional development, rich instructor input, and acknowledgement 
of areas of strength or weakness. The self-reviews are not viewed by the evaluator during 
the term, although the instructor can choose to share the self-reviews with the evaluator if 
deemed appropriate to enhance understanding or meet mentoring needs. The self-reviews 
are compiled via a secure online portal and shared with the academic department at the 
end of the term. 

The summative component of the OIES includes a summative review and end-of-term 
student evaluations of the course and instructor. The summative review is completed by 
the same evaluator who conducted the formative/mentoring process during the term. The 
summative review is designed to encapsulate the formative review process and to stress 
changes (either positive or negative) in the instructor’s facilitation of the course during the 
formative/mentoring phase. Ultimately, the evaluator makes a recommendation of wheth¬ 
er the instructor should be a) retained, b) retained with contingencies, or c) not retained 
as an online instructor at the university. A fourth level, probation, was used briefly, but this 
category was difficult to distinguish from retain with contingencies. For this reason the two 
ratings were merged. As remediation and retraining are desirable for most online instruc¬ 
tors having difficulty, the merged category used the label retain with contingencies to con¬ 
note a more mentoring, developmental stance by the university. While the overall rating 
levels provide categorization and overall feedback, instructors and evaluators suggest that 
it is the comments (in both formative and summative reviews) that provide the most helpful 
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guidance to online instructors. 

The summative components of the OIES and instructor self-reviews are delivered to the 
instructor’s academic department online program coordinator. Based upon the summative 
review, instructor self-reviews, and student evaluations, the program coordinator deter¬ 
mines which individual courses the instructor may be assigned to teach in the future or if 
the instructor no longer will be given departmental teaching assignments. Online admin¬ 
istrators also use the summative components of the OIES to determine if the instructor is 
adept at handling online instruction and the particular policies of online learning. In this 
way, both the academic and administrative areas utilize the OIES when making instructor 
retention decisions. 

Outcomes of the OIES 

Of the 437 OIES evaluations conducted to date, 379 (86%) bore the final rating of retain. 
Thirty-eight reviews were rated retain with contingencies, nine were probation (a term 
which is not currently used), and 11 were in the category of do not retain. These statistics 
support the overall intent of the OIES as a mentoring mechanism to retain and retrain 
online instructors. Because so few instructors received negative ratings, we concluded that 
the OIES either reinforced existing positive online facilitation or that instructors who may 
have been struggling in their online facilitation were properly guided and mentored via 
the formative reviews so as to result in acceptable improvement by the end of the process. 
One cannot assume that the OIES itself leads to good online facilitation, but the data and 
outcomes of the OIES do enhance instructor awareness of and adherence to policy and best 
practices in online learning. 

After implementing the OIES, the team found numerous strengths and some weaknesses. 
These strengths and weaknesses fell into two main categories: administration-oriented and 
instructor-oriented. Administrative issues included a) the time involved in completing each 
formative review, b) standardizing the nomenclature and comments that passed between 
the instructor evaluators, and c) managing the list of current and future reviewees for the 
OIES. Instructor-oriented issues included a) explaining and allowing for differences in the 
instructional strategies used across courses (that is, the instructor presentation of devel¬ 
oped course content) and b) the notable differences between new instructors and experi¬ 
enced instructors regarding their perceptions of the OIES. 

Time 

Early in the implementation of the OIES, the instructor evaluators discovered that the for¬ 
mative reviews (which were completed and delivered to instructors every two weeks) were 
very time-consuming. Each criterion on the reviews required the instructor evaluator to 
access the online course via the learning management system and to scrutinize multiple 
areas in which the instructor facilitated student learning. This process became even more 
time-consuminG as the instructor evaluators often had to compare instructor facilitation of 
course content to the master course developed content. If they found instructor deviations 
from the master course content (which could be either beneficial or detrimental depending 
upon the deviation), the evaluator had to spend additional time consulting with adminis- 
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trators or the academic department concerning the content changes. It was not uncommon 
for a normal formative review to take 30-50 minutes to complete. One strategy to address 
this time factor was for each evaluator to evaluate a group of instructors teaching differ¬ 
ent sections of the same course. This strategy allowed the instructor evaluator to become 
very familiar with the developed content and to then discern efficiently the differences each 
instructor used in the facilitation of their course section. Consequently, evaluating several 
sections of the same course reduced the time required to approximately 20-35 minutes per 
review. 

Standard Language 

Another administrative issue involved standardizing the language used by the instructor 
evaluators. This element was a natural extension of the criterion-based nature of the for¬ 
mative reviews. The instructor evaluators had to ensure that their interpretation of the cri¬ 
teria was consistent not only among themselves, but also with the policies of the online 
operations unit and academic departments. Each instructor evaluator spent much time in 
telephonic and email communication with the other instructor evaluators to ensure that 
a uniform message would be conveyed to online instructors. Building on this desire for 
uniformity of interpretation, the OIES team developed comment banks for each review. 
The comment banks were shared with each instructor evaluator and helped ensure that 
each online instructor received the same information about Park policies and online in¬ 
structor facilitation expectations. Instructor evaluators did have the latitude to customize 
their comments for each review and for each instructor, but the comment banks increased 
continuity and equity in the formative reviews. The comment banks were also time-efficient 
as the same comment, if warranted, could be quickly cut and pasted into several formative 
reviews. Another strategy for achieving standard language among the evaluators would be 
interrater reliability research. Interrater reliability has not yet been assessed for the OIES, 
but it would be an appropriate way to gauge standardization and equitable treatment in 
reviews. 

Management 

A final administrative consideration was managing a list of current and future OIES re- 
viewees. To meet this need, a spreadsheet was created and archived by the lead instructor 
evaluator. The spreadsheet included all the pertinent administrative information as well 
as OIES term, evaluator, and summative evaluation information. In addition, this archival 
system provided a plan for future OIES terms by anticipating instructor teaching assign¬ 
ments and evaluator availability. The lead instructor evaluator kept track of online instruc¬ 
tors who received retain with contingencies summative evaluation ratings and planned for 
these instructors to be evaluated again during the next term. A system of OIES referral was 
also created in which operations staff and department program coordinators could suggest 
instructors for OIES evaluation. Referrals could be made on the grounds of administrative 
infractions (failure to follow university policies) or facilitation concerns (improper interac¬ 
tion with students in their online classroom). While it was a sufficient administrative tool, 
the master spreadsheet and working versions became unwieldy and prone to input error. 
Attempts were made to create a database and online portal entry for administrative needs, 
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but these needs were not met due to a lack of computer programming personnel and/or 
funding resources. 

Course/Instructor Differences 

Instructor-oriented issues with the OIES were more revealing, arguably, than the admin¬ 
istrative issues. A perennial online instructor issue related to the difference between an 
instructor and a course developer. According to university policy, course developers cre¬ 
ate the course content, following university and operations guidelines; and this content is 
subject to approval by the academic department. The developed material is then provided 
to the various online instructors for individual course sections. Instructors are encouraged 
to add supplemental course content to provide for their own instructional differences and 
preferences. This policy was confusing to online instructors; therefore, the OIES served as 
a reinforcement of this guideline. Some instructors felt their creativity was stifled by the 
policy, but the mentoring exchanges with the instructor evaluators provided suggestions 
for taking the developed course content and enhancing the material with their own facili¬ 
tation methods. Through the formative reviews and mentoring early in the term, instruc¬ 
tors received guidance and tips from the evaluator to enhance the later weeks of the term. 
Evaluators were able to explain university policy and expectations in a real course context. 
In some instances, evaluators served as a peer bridge between content developers and indi¬ 
vidual section instructors. The presence of an intermediary was also beneficial when misin¬ 
terpretations occurred between administrative needs and instructor intentions. 

New versus Experienced Instructors 

One striking difference was the perception of the OIES among new instructors compared 
to existing instructors. The OIES was envisioned primarily as an efficient mentoring tool 
to aid new online instructors on the grounds that new instructors, unfamiliar to university 
policies and/or online instruction, would be the greatest beneficiaries of the OIES forma¬ 
tive reviews. It was also believed that these new instructors would be suspect of the OIES 
as a “judgment” of their online facilitation ability. In actuality, the instructor evaluators 
observed that new instructors were among the most receptive to the formative, mentoring 
reviews. They appreciated the guidance and even the way in which the formative reviews 
provided them with a measure of their online facilitation performance during the term. 
The early formative reviews were timed so that corrections could be made within the ac¬ 
tive term in order to benefit current students. One new instructor commented, “I love the 
constructive criticism and since this is my first time teaching online courses, it is greatly 
appreciated.” Another instructor echoed, “This being my first course online has been a great 
experience, learning as I go as well as generating ideas for me on how to make changes in 
the course materials, supplements, etc.” 

Experienced Park online instructors were equally appreciative of the OIES, but the in¬ 
structor evaluators witnessed an initial suspicion from these instructors. These suspicions 
ranged from questions about a) why they were being evaluated, b) how the information 
would be reported to their department, and c) the credentials of the instructor evaluators. 
The instructor evaluators found that it was best to quell these concerns by underscoring the 
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fact that the OIES was primarily a formative, mentoring evaluation system. The reviews 
were designed to guide and suggest, not judge and dictate. As these existing instructors 
became familiar with the review process and the mentoring discussions with the instructor 
evaluator, they too became appreciative of the OIES and looked to the instructor evaluators 
for new ideas to enhance their teaching. The following is a good example of this mentality 
of sharing, fostered by the OIES. “Are there examples to share from other instructors as 
to how they might be embellishing the rubric, grading system if that’s what you want us 
to do?” A common culminating comment to an instructor evaluator from an experienced 
instructor was, “Thanks for your excellent suggestions and mentoring. They were very ben¬ 
eficial.” This was the predominant sentiment from existing and new instructors after expe¬ 
riencing the OIES. 


Conclusions 


Evaluation is a human process. As such, the OIES promotes the evaluator’s ability to work 
effectively to positively guide an instructor during the course. Cognizant of issues that can 
arise (i.e., instructor illness, course room development issues, technological challenges, 
natural disasters, or other life issues) the evaluator also must provide some room for flex¬ 
ibility. The instructor stands central in this evaluation process and is assessed distinctly 
with a final observation rating or outcome level. Regardless of the outcome of the OIES 
evaluation in terms of the instructor’s rating, the continual conversation centers on estab¬ 
lishing the ideal environmental conditions online for students to forge learning and criti¬ 
cal thinking. Through integrated support, a learning community online, and ongoing pro¬ 
fessional development resources with examples that identify best practices, the instructor 
finds collegial support. 

Overall, the instructor evaluation team was encouraged by the instructor reactions to the 
OIES. The team realized that clarification and emphasis on the formative/mentoring na¬ 
ture of the reviews was extremely important. It was equally important to spend as much 
time as necessary in the mentoring dialogues between the instructor and instructor evalu¬ 
ator. The reviews proved to be an excellent guide for these mentoring exchanges, which 
occur within a context of guidance and mutual benefit. However, at times the evaluators 
experienced frustration when their mentoring advice was ignored. 

The evaluation component became problematic for in the final analysis, at close of term, the 
instructors’ skills, progress, and potential must be categorized in a final recommendation 
in order to meet administrative needs of the institution. Evaluators experienced conflict 
of interest when shifting from a mentoring role to that of evaluator. Rendering these de¬ 
terminations often severed the mentoring relationship permanently. The team concluded 
that mixing mentoring with a high-stakes judgment is illogical. Therefore, future evaluative 
mechanisms at Park University separated the mentoring and evaluation functions. 

Another charge from the university pertained to an annual teaching evaluation. The thor¬ 
ough, nurturing nature of the OIES reviews placed severe constraints on the number of on¬ 
line instructors who could be evaluated annually by the limited pool of available evaluators. 
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There were simply too many instructors and too few evaluators. Fulfilling the annual evalu¬ 
ation requirement for all online adjunct instructors was unsustainable given the mentor¬ 
ing nature of the reviews. Knowing that no more evaluators were available, a streamlined 
evaluation model was sought. 

A shorter instrument was devised that focused only on evaluation. Th e faculty online ob¬ 
servation (FOO) reflected the lessons learned from implementing the OIES. The FOO con¬ 
sists of fewer evaluation criteria, observes an instructor during a short, finite span of time, 
and yet retains the necessary online evaluation components stressed in the research litera¬ 
ture and found to be paramount in the OIES experience. The FOO continues to emphasize 
the same critical areas, thereby ensuring that student learning needs are still met via proper 
online instructor facilitation. The FOO uses the same retrievable archive as the OIES for 
university administrative use and instructor feedback. 

Every aspect of online education at Park University occurs collaboratively. From situat¬ 
ing a course within a curriculum, through syllabus formation, course design, development, 
delivery, approvals, to faculty training, we anticipate and solve problems relative to qual¬ 
ity. Every online instructor is welcomed to Park with the understanding that providing the 
best educational experience available is our top priority. The next logical step is to ensure 
that the instructor possesses the proper skills, mindset, and expertise to facilitate student 
learning. The OIES provided valuable insight into the hazards of blending mentoring and 
evaluations. The FOO brings to fruition these high standards and the demands of modern 
distance education. 
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